Eye tracking evidence for the reinstatement of emotionally negative and neutral memories.

Recent eye tracking studies have linked gaze reinstatement-when eye movements from encoding are reinstated during retrieval-with memory performance. In this study, we investigated whether gaze reinstatement is influenced by the affective salience of information stored in memory, using an adaptation of the emotion-induced memory trade-off paradigm. Participants learned word-scene pairs, where scenes were composed of negative or neutral objects located on the left or right side of neutral backgrounds. This allowed us to measure gaze reinstatement during scene memory tests based on whether people looked at the side of the screen where the object had been located. Across two experiments, we behaviorally replicated the emotion-induced memory trade-off effect, in that negative object memory was better than neutral object memory at the expense of background memory. Furthermore, we found evidence that gaze reinstatement was related to recognition memory for the object and background scene components. This effect was generally comparable for negative and neutral memories, although the effects of valence varied somewhat between the two experiments. Together, these findings suggest that gaze reinstatement occurs independently of the processes contributing to the emotion-induced memory trade-off effect.

Improving Memory Search Through Model-Based Cue Selection.

We often use cues from our environment when we get stuck searching our memories, but prior research has failed to show benefits of cuing with other, randomly selected list items during memory search. What accounts for this discrepancy? We proposed that cues' content critically determines their effectiveness and sought to select the right cues by building a computational model of how cues affect memory search. Participants (N = 195 young adults from the United States) recalled significantly more items when receiving our model's best (vs. worst) cue. Our model provides an account of why some cues better aid recall: Effective cues activate contexts most similar to the remaining items' contexts, facilitating recall in an unsearched area of memory. We discuss our contributions in relation to prominent theories about the effect of external cues.

When working memory may be just working, not memory.

The N-back task is often considered to be a canonical example of a task that relies on working memory (WM), requiring both maintenance of representations of previously presented stimuli and also processing of these representations. In particular, the set-size effect in this task (e.g., poorer performance on three-back than two-back judgments), as in others, is often interpreted as indicating that the task relies on retention and processing of information in a limited-capacity WM system. Here, we consider an alternative possibility: that retention in episodic memory (EM) rather than WM can account for both set-size and lure effects in the N-back task. Accordingly, performance in the N-back task may reflect engagement of the processing ("working") function of WM but not necessarily limits in either that processing ability nor in retention ("memory"). To demonstrate this point, we constructed a neural network model that was augmented with an EM component, but lacked any capacity to retain information across trials in WM, and trained it to perform the N-back task. We show that this model can account for the set-size and lure effects obtained in an N-back study by M. J. Kane et al. (2007), and that it does so as a result of the well-understood effects of temporal distinctiveness on EM retrieval, and the processing of this information in WM. These findings help illuminate the ways in which WM may interact with EM in the service of cognitive function and add to a growing body of evidence that tasks commonly assumed to rely on WM may alternatively (or additionally) rely on EM. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Optimal metacognitive control of memory recall.

Most of us have experienced moments when we could not recall some piece of information but felt that it was just out of reach. Research in metamemory has established that such judgments are often accurate; but what adaptive purpose do they serve? Here, we present an optimal model of how metacognitive monitoring (feeling of knowing) could dynamically inform metacognitive control of memory (the direction of retrieval efforts). In two experiments, we find that, consistent with the optimal model, people report having a stronger memory for targets they are likely to recall and direct their search efforts accordingly, cutting off the search when it is unlikely to succeed and prioritizing the search for stronger memories. Our results suggest that metamemory is indeed adaptive and motivate the development of process-level theories that account for the dynamic interplay between monitoring and control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

INDUCING REPRESENTATIONAL CHANGE IN THE HIPPOCAMPUS THROUGH REAL-TIME NEUROFEEDBACK.

When you perceive or remember one thing, other related things come to mind. This competition has consequences for how these items are later perceived, attended, or remembered. Such behavioral consequences result from changes in how much the neural representations of the items overlap, especially in the hippocampus. These changes can reflect increased (integration) or decreased (differentiation) overlap; previous studies have posited that the amount of coactivation between competing representations in cortex determines which will occur: high coactivation leads to hippocampal integration, medium coactivation leads to differentiation, and low coactivation is inert. However, those studies used indirect proxies for coactivation, by manipulating stimulus similarity or task demands. Here we induce coactivation of competing memories in visual cortex more directly using closed-loop neurofeedback from real-time fMRI. While viewing one object, participants were rewarded for implicitly activating the representation of another object as strongly as possible. Across multiple real-time fMRI training sessions, they succeeded in using the neurofeedback to induce coactivation. Compared with untrained objects, this coactivation led to behavioral and neural integration: The trained objects became harder for participants to discriminate in a categorical perception task and harder to decode from patterns of fMRI activity in the hippocampus.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening.

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Associative memory retrieval modulates upcoming perceptual decisions.

Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant's responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.

Differentiation and Integration of Competing Memories: A Neural Network Model.

What determines when neural representations of memories move together (integrate) or apart (differentiate)? Classic supervised learning models posit that, when two stimuli predict similar outcomes, their representations should integrate. However, these models have recently been challenged by studies showing that pairing two stimuli with a shared associate can sometimes cause differentiation, depending on the parameters of the study and the brain region being examined. Here, we provide a purely unsupervised neural network model that can explain these and other related findings. The model can exhibit integration or differentiation depending on the amount of activity allowed to spread to competitors - inactive memories are not modified, connections to moderately active competitors are weakened (leading to differentiation), and connections to highly active competitors are strengthened (leading to integration). The model also makes several novel predictions - most importantly, that differentiation will be rapid and asymmetric. Overall, these modeling results provide a computational explanation for a diverse set of seemingly contradictory empirical findings in the memory literature, as well as new insights into the dynamics at play during learning.

Large language models can segment narrative events similarly to humans.

Humans perceive discrete events such as "restaurant visits" and "train rides" in their continuous experience. One important prerequisite for studying human event perception is the ability of researchers to quantify when one event ends and another begins. Typically, this information is derived by aggregating behavioral annotations from several observers. Here we present an alternative computational approach where event boundaries are derived using a large language model, GPT-3, instead of using human annotations. We demonstrate that GPT-3 can segment continuous narrative text into events. GPT-3-annotated events are significantly correlated with human event annotations. Furthermore, these GPT-derived annotations achieve a good approximation of the "consensus" solution (obtained by averaging across human annotations); the boundaries identified by GPT-3 are closer to the consensus, on average, than boundaries identified by individual human annotators. This finding suggests that GPT-3 provides a feasible solution for automated event annotations, and it demonstrates a further parallel between human cognition and prediction in large language models. In the future, GPT-3 may thereby help to elucidate the principles underlying human event perception.

Evidence That Event Boundaries Are Access Points for Memory Retrieval.

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Optimal policies for free recall.

There is rich structure in the order in which studied material is recalled in a free recall task (Howard & Kahana, 2002a). Extensive effort has been directed at understanding the processes and representations that give rise to this structure; however, it remains unclear why certain types of recall organization might be favored in the first place. We provide a rational analysis of the free recall task, deriving the optimal policy for recalling items under the internal representations and processes described by the context maintenance and retrieval (CMR) model of memory search (Polyn et al., 2009a). Our model, which we call rational-CMR, shows that the optimal policy for free recall is to start from the beginning of the list and then sequentially recall forwards, providing a rational account of the primacy and forward asymmetry effects typically observed in free recall. In addition, when recall is not initiated from the beginning of list, it is optimal during recall transitions to minimize the amount of forward asymmetry. Predictions from the rational model are confirmed in human behavioral data: Top-performing human participants demonstrate a stronger tendency to initiate recall from the beginning of the list and carry forward recalls, and the amount of forward asymmetry in participants depends on whether they start recall from the beginning or end of the list. We discuss the resemblance of optimal behavior in free recall to participants' behavior when applying mnemonic techniques such as the method of loci. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Neural representations of naturalistic events are updated as our understanding of the past changes.

The brain actively reshapes our understanding of past events in light of new incoming information. In the current study, we ask how the brain supports this updating process during the encoding and recall of naturalistic stimuli. One group of participants watched a movie ('The Sixth Sense') with a cinematic 'twist' at the end that dramatically changed the interpretation of previous events. Next, participants were asked to verbally recall the movie events, taking into account the new 'twist' information. Most participants updated their recall to incorporate the twist. Two additional groups recalled the movie without having to update their memories during recall: one group never saw the twist; another group was exposed to the twist prior to the beginning of the movie, and thus the twist information was incorporated both during encoding and recall. We found that providing participants with information about the twist beforehand altered neural response patterns during movie-viewing in the default mode network (DMN). Moreover, presenting participants with the twist at the end of the movie changed the neural representation of the previously-encoded information during recall in a subset of DMN regions. Further evidence for this transformation was obtained by comparing the neural activation patterns during encoding and recall and correlating them with behavioral signatures of memory updating. Our results demonstrate that neural representations of past events encoded in the DMN are dynamically integrated with new information that reshapes our understanding in natural contexts.

A model of autonomous interactions between hippocampus and neocortex driving sleep-dependent memory consolidation.

How do we build up our knowledge of the world over time? Many theories of memory formation and consolidation have posited that the hippocampus stores new information, then "teaches" this information to the neocortex over time, especially during sleep. But it is unclear, mechanistically, how this actually works-How are these systems able to interact during periods with virtually no environmental input to accomplish useful learning and shifts in representation? We provide a framework for thinking about this question, with neural network model simulations serving as demonstrations. The model is composed of hippocampus and neocortical areas, which replay memories and interact with one another completely autonomously during simulated sleep. Oscillations are leveraged to support error-driven learning that leads to useful changes in memory representation and behavior. The model has a non-rapid eye movement (NREM) sleep stage, where dynamics between the hippocampus and neocortex are tightly coupled, with the hippocampus helping neocortex to reinstate high-fidelity versions of new attractors, and a REM sleep stage, where neocortex is able to more freely explore existing attractors. We find that alternating between NREM and REM sleep stages, which alternately focuses the model's replay on recent and remote information, facilitates graceful continual learning. We thus provide an account of how the hippocampus and neocortex can interact without any external input during sleep to drive useful new cortical learning and to protect old knowledge as new information is integrated.

Real-time neurofeedback to alter interpretations of a naturalistic narrative.

We explored the potential of using real-time fMRI (rt-fMRI) neurofeedback training to bias interpretations of naturalistic narrative stimuli. Participants were randomly assigned to one of two possible conditions, each corresponding to a different interpretation of an ambiguous spoken story. While participants listened to the story in the scanner, neurofeedback was used to reward neural activity corresponding to the assigned interpretation. After scanning, final interpretations were assessed. While neurofeedback did not change story interpretations on average, participants with higher levels of decoding accuracy during the neurofeedback procedure were more likely to adopt the assigned interpretation; additional control conditions are needed to establish the role of individualized feedback in driving this result. While naturalistic stimuli introduce a unique set of challenges in providing effective and individualized neurofeedback, we believe that this technique holds promise for individualized cognitive therapy.

RT-Cloud: A cloud-based software framework to simplify and standardize real-time fMRI.

Real-time fMRI (RT-fMRI) neurofeedback has been shown to be effective in treating neuropsychiatric disorders and holds tremendous promise for future breakthroughs, both with regard to basic science and clinical applications. However, the prevalence of its use has been hampered by computing hardware requirements, the complexity of setting up and running an experiment, and a lack of standards that would foster collaboration. To address these issues, we have developed RT-Cloud (https://github.com/brainiak/rt-cloud), a flexible, cloud-based, open-source Python software package for the execution of RT-fMRI experiments. RT-Cloud uses standardized data formats and adaptable processing streams to support and expand open science in RT-fMRI research and applications. Cloud computing is a key enabling technology for advancing RT-fMRI because it eliminates the need for on-premise technical expertise and high-performance computing; this allows installation, configuration, and maintenance to be automated and done remotely. Furthermore, the scalability of cloud computing makes it easier to deploy computationally-demanding multivariate analyses in real time. In this paper, we describe how RT-Cloud has been integrated with open standards, including the Brain Imaging Data Structure (BIDS) standard and the OpenNeuro database, how it has been applied thus far, and our plans for further development and deployment of RT-Cloud in the coming years.

Schema representations in distinct brain networks support narrative memory during encoding and retrieval.

Schematic prior knowledge can scaffold the construction of event memories during perception and also provide structured cues to guide memory search during retrieval. We measured the activation of story-specific and schematic representations using fMRI while participants were presented with 16 stories and then recalled each of the narratives, and related these activations to memory for specific story details. We predicted that schema representations in medial prefrontal cortex (mPFC) would be correlated with successful recall of story details. In keeping with this prediction, an anterior mPFC region showed a significant correlation between activation of schema representations at encoding and subsequent behavioral recall performance; however, this mPFC region was not implicated in schema representation during retrieval. More generally, our analyses revealed largely distinct brain networks at encoding and retrieval in which schema activation was related to successful recall. These results provide new insight into when and where event knowledge can support narrative memory.

Our day-to-day experiences are incredibly complex, so how does the brain remember them? Cognitive scientists have shown that memories rely on knowledge of common events that we have experienced before. Think about going to a restaurant: you arrive, you find a table, you order food, and then you eat. This kind of predictable sequence is called a schema. When humans make memories, our brains use schemas like these as scaffolding. They take a basic pattern constructed from past experience and fill it in with the specific details of an event. When memories are recalled, our brains use schemas as step-by-step guides to remember the events in the right order. Most research so far on how the brain uses schemas for memory has involved showing participants pictures or words and then testing their memory by asking 'true or false' questions. This revealed that a brain area called the medial prefrontal cortex plays an important role in creating and retrieving memories for items related to a schema. But, studies have not yet assessed exactly how the brain uses schemas to understand and remember a long, realistic event that unfolds over several minutes. To answer this question, Masís-Obando et al. scanned people's brains while they watched or listened to clips of two familiar experiences: eating at a restaurant or catching a flight at an airport. Then, the participants were scanned while they tried to retell each story in their own words. The volunteers were graded based on how many details they recalled. The scans showed that when volunteers' medial prefrontal cortex kept track of the schema throughout the whole time that an event was happening, they were more likely to score well on the memory test. But it wasn't necessary for medial prefrontal cortex to hold the schema in mind when remembering the story. Instead, a different set of brain regions maintained schema information during successful remembering. This study reveals new information about how memories and schemas work that could help explain why people develop problems making or recalling memories in diseases such as Alzheimer's. The findings could also be used to help people make experiences or stories more memorable.

Shared computational principles for language processing in humans and deep language models.

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

A neural network model of when to retrieve and encode episodic memories.

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

The human brain can record snapshots of details from specific events ­ such as where and when the event took place ­ and retrieve this information later. Recalling these 'episodic memories' can help us gain a better understanding of our current surroundings and predict what will happen next. Studies of episodic memory have typically involved observing volunteers while they perform simple, well-defined tasks, such as learning and recalling lists of random pairs of words. However, it is less clear how episodic memory works 'in the wild' when no one is quizzing us, and we are going about everyday activities. Recently, researchers have started to study memory in more naturalistic situations, for example, while volunteers watch a movie. Here, Lu et al. have built a computational model that can predict when our brains store and retrieve episodic memories during these experiments. The team gave the model a sequence of inputs corresponding to different stages of an event, and asked it to predict what was coming next. Intuitively, one might think that the best use of episodic memory would be to store and retrieve snapshots as frequently as possible. However, Lu et al. found that the model performed best when it was more selective ­ that is, preferentially storing episodic memories at the end of events and waiting to recover them until there was a gap in the model's understanding of the current situation. This strategy may help the brain to avoid retrieving irrelevant memories that might (in turn) result in the brain making incorrect predictions with negative outcomes. This model makes it possible for researchers to predict when the brain may store and retrieve episodic memories in a particular experiment. Lu et al. have openly shared the code for the model so that other researchers will be able to use it in their studies to understand how the brain uses episodic memory in everyday situations.

Increasing stimulus similarity drives nonmonotonic representational change in hippocampus.

Studies of hippocampal learning have obtained seemingly contradictory results, with manipulations that increase coactivation of memories sometimes leading to differentiation of these memories, but sometimes not. These results could potentially be reconciled using the nonmonotonic plasticity hypothesis, which posits that representational change (memories moving apart or together) is a U-shaped function of the coactivation of these memories during learning. Testing this hypothesis requires manipulating coactivation over a wide enough range to reveal the full U-shape. To accomplish this, we used a novel neural network image synthesis procedure to create pairs of stimuli that varied parametrically in their similarity in high-level visual regions that provide input to the hippocampus. Sequences of these pairs were shown to human participants during high-resolution fMRI. As predicted, learning changed the representations of paired images in the dentate gyrus as a U-shaped function of image similarity, with neural differentiation occurring only for moderately similar images.

High-Order Areas and Auditory Cortex Both Represent the High-Level Event Structure of Music.

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.